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(54) Image space display method and apparatus 

(57) An image space display method facilitates a us- 
er to grasp a feature space by assigning each feature 
to a respective one of dimensional axes of a display 
space. The image-space display method extracts fea- 
tures from images, hierarchically divides a feature 
space of the features, virtually converts the images into 
a tree structure, divides a display space according to the 



tree-structure, and displays the image space by display- 
ing the Images on each of the divided display spaces. 
In the method a tree-structure is generated for each of 
the features. Dimension data corresponding to a 
number of the features is generated by mapping each 
tree structure in one-dimension. The dimension data is 
displayed on the corresponding divided display spaces 
as display coordinate-axis data. 



nG.2 



11 



DIGITAL CAMERA 
SCANNER 



CM 
< 

CM 

to 

CM 



MONITOR 



1. 



IMAGE APPUCATION 



1 




81 




FEATURE-EXTRACTION 

UNrr 








82 




CLUSTERING UNIT 










TREE-STRUCTURE 
GENERATION UNIT 










DISPLAY-SPACE 
GENERATION UNIT 






r 85 




DISPLAY-SCREEN 
GENERATION UNIT 


















8 



Q. 

LU 



Printed by Jouve. 75001 PARIS (FR) 



1 



EP1 246 124 A2 



2 



Description 

[0001] The present invention generally relates to a 
technique for displaying an image space and, more par- 
ticularly, to an image space display method and appa- 
ratus for spatially displaying features of image. 
[0002] As for such a technique for displaying an image 
space, there are following techniques: 

1) "A User Interface Visualizing feature Space for 
Content- Based Image Retrieval", Technical Report 
of lEICE, IE98-204; and 

2) "Visual Interaction for Exploration in Information 
Space of Documents", Journal of JSSST, Vol.13. 

[0003] The above-mentioned known technique 1 ) is a 
method for mapping image features on a display space 
of a 2-dimensional by reducing the number of dimen- 
sions of a multidimensional space according to principal 
component analysis. The above-mentioned known 
technique 2) introduces an Idea of dynamic updating of 
visualized results in a visual classification technique 
where the updating is performed In response to a user 
operation, and the visual classification is given by ar- 
ranging a large number of documents and keywords 
based on their mutual relationships. 
[0004] However, the known technique 1 ) has a prot>- 
lem in that principal component analysis cannot be car- 
ried out when image features cannot be represented by 
vector data or when image similarity between images 
cannot be represented by a linear function. The known 
technique 2) has a drawback in that the number of key- 
words becomes large due to a large amount of compu- 
tation, which makes a processing time lengthy, thereby 
making it unsuitable for interactive presentation. 
[0005] On the other hand, when a plurality of features 
area extracted from an Image so as to display the fea- 
ture space thereof on a 3-dimensinal or 3-dimensional 
display space, a user can easily grasp the feature space 
if the features are assigned to the respective dimension- 
al axes so as to give sense to each dimensional axis. 
[0006] It is a general object of the present invention 
to provide an improved and useful image space display 
method and apparatus in which the above-mentioned 
problems are eliminated. 

[0007] A more specific object of the present Invention 
is to provide an image space display method and appa- 
ratus which facilitates a user to grasp a feature space 
by assigning each feature to a respective one of dimen- 
sional axes of a display space. 
[0008] In order to achieve the above-mentioned ob- 
jects, there is provided according to one aspect of the 
present invention an image-space display method of ex- 
tracting features from images, hierarchically dividing a 
feature space of the features, virtually converting the im- 
ages into a tree structure, dividing a display space ac- 
cording to the tree-structure, and displaying the image 
space by displaying the images on each of the divided 



display spaces, the method further comprising the steps 
of: generating a tree-structure for each of the features; 
generating dimension data corresponding to a number 
of the features by mapping each tree structure in one- 
5 dimension; and displaying the dimension data on the 
corresponding divided display spaces as display coor- 
dinate-axis data. 

[0009] According to the present invention, when a plu- 
rality of features are used, the tree-structure Is generat- 

10 ed for each of the features and each tree structure is 
mapped in one-dimension so as to dimensional data 
corresponding to the number of the features. The di- 
mensional data is used as the display coordinate-axes 
data so as to map the features in the dimensions of the 

15 display space. Thus, the image space displayed by the 
method of the present enables a user to easily grasp the 
feature space. 

[0010] In the image space display method according 
to the present invention, dividing the display space may 

20 include the step of: obtaining distances between each 
point of the features and center points of two clusters 
with respect to each point; sorting each points according 
to a difference between the obtained distances; and di- 
viding the display space into two clusters by setting a 

25 boundary of the clusters according to an order of the 
sorting.* 

[0011] The boundary of the clusters may be set be- 
tween points of which a difference of distances preced- 
ing and proceeding points is largest in the order of the 
30 sorting. A area corresponding to one of dimensions of 
the display space may be divided in accordance with 
distances to center points of the two clusters and a ratio 
of differences of the difference distances at the bound- 
ary of the clusters. The boundary of the clusters may 
35 correspond to a halfway point in the order of the sorting. 
The features may be three-dimensional data con^e- 
sponding to form, texture and color of the images, and 
the features are displayed by using the three-dimen- 
sional data as display coordinate-axes data. 
40 [001 2] Additionally, there is provided according to an- 
other aspect of the present invention an image-space 
display method of extracting features from Images, hi- 
erarchically dividing a feature space of the features, vir- 
tually converting the images into a tree structure, divid- 
es ing a display space according to the tree-structure, and 
displaying the image space by displaying the images on 
each of the divided display spaces, the method further 
comprising the steps of: assigning the extracted fea- 
tures to respective dimensional axes of the display 
50 space; dividing recursively the feature space into four 
sub-spaces in accordance with the dimensional axes 
while reflecting a relationship between the images ar- 
ranged in a two-dimensional display space in a relation- 
ship between the images on the feature space; and lo- 
ss eating the tree-structure of each sub-space in four sub- 
spaces generated by dividing the two-dimensional dis- 
play space. 

[0013] According to the above-mentioned invention. 
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an image feature space display process for mapping 
each feature on the dimensional axes of the display 
space can be performed with high accuracy, and the im- 
age space displayed by the method of the above-men- 
tioned invention enables a user to easily grasp the fea- 
ture space. 

[0014] In the image space display method according 
to the present invention, when selecting a main image 
of each sub-space in the step of dividing into four sub- 
spaces, the main image may be sequentially deter- 
mined based on a result of calculation according to an 
evaluation equation that is previously established based 
on a distance to main image previously determined in 
consideration with a positional relationship with respect 
to each feature. 

[0015] Additionally, the once determined main Image 
may be recalculated based on evaluation equations 
based on three other main Images so as to determine 
the main image again based on a result of the recalcu- 
lation. A position of the main Image may be calculated 
repeatedly until a change In the position does not occur, 
and the position at which a change does not occur is 
selected as the position of the main Image. When a 
number of images assigned to the closest main image 
exceeds a maximum number of images containable In 
the sub-space containing the closest main image, a 
process of removing one of the Images farthest to the 
main image from the sub-space concerned may be re- 
peatedly performed so as to distribute the Images uni- 
formly to the sub spaces. A three-dimensional display 
space may be used Instead of the two-dimensional dis- 
play space. 

[0016] Additionally, there is provided another aspect 
of the present invention an image-space display appa- 
ratus for extracting features from Images, hierarchically 
dividing a feature space of the features, virtually con- 
verting the images into a tree structure, dividing a dis- 
play space according to the tree-structure, and display- 
ing the image space by displaying the Images on each 
of the divided display spaces, the apparatus comprising: 
means for generating a tree-structure for each of the 
features; means for generating dimension data corre- 
sponding to a number of the features by mapping each 
tree structure In one-dimension; and means for display- 
ing the dimension data on the corresponding divided 
display spaces as display coordinate-axis data. 
[001 7] Additionally, there is provided according to an- 
other aspect of the present invention an image-space 
display apparatus for extracting features from images, 
hierarchically dividing a feature space of the features, 
virtually converting the Images into a tree structure, di- 
viding a display space according to the tree-structure, 
and displaying the image space by displaying the imag- 
es on each of the divided display spaces, the apparatus 
comprising: means for assigning the extracted features 
to respective dimensional axes of the display space; 
means for dividing recursively the feature space into 
four sub-spaces in accordance with the dimensional ax- 



es while reflecting a relationship between the linages 
arranged in a two-dimensional display space In a rela- 
tionship between the images on the feature space; and 
means for locating the tree-structure of each sub-space 
5 In four sub-spaces generated by dividing the two-dimen- 
sional display space. 

[001 8] Additionally, there Is provided according to an- 
other aspect of the present Invention a computer-read- 
able medium having a program embodied therein for 
10 causing a computer to extract features from images, hi- 
erarchically divide a feature space of the features, vir- 
tually convert the images into a tree structure, divide a 
display space according to the tree-structure, and dis- 
play the image space by displaying the images on each 
15 of the divided display spaces, said program comprising: 
a program code for generating a tree-structure for each 
of the features; a program code for generating dimen- 
sion data corresponding to a number of the features by 
mapping each tree structure in one-dimension; and a 
program code for displaying the dimension data on the 
corresponding divided display spaces as display coor- 
dinate-axis data. 

[001 9] Additionally, there is provided according to an- 
other aspect of the present invention a computer-read- 
able medium having a program embodied therein for 
causing a computer to extract features from images, hi- 
erarchically divide a feature space of the features, vir- 
tually convert the images Into a tree structure, divide a 
display space according to the tree-structure, and dis- 
play the image space by displaying the images on each 
of the divided display spaces, said program comprising: 
a program code for assigning the extracted features to 
respective dimensional axes of the display space; a pro- 
gram code for dividing recursively the feature space Into 
four sub-spaces in accordance with the dimensional ax- 
es while reflecting a relationship between the images 
arranged In a two-dimensional display space in a rela- 
tionship between the images on the feature space; and 
a program code for locating the tree-structure of each 
sub-space In four sub-spaces generated by dividing the 
two-dimensional display space. 
[0020] Other objects, features and advantages of the 
present Invention will become more apparent from the 
following detailed description when read In conjunction 
with the accompanying drawings. 

FIG. 1 Is a block diagram of an image display ap- 
paratus according to a first embodiment of the 
present Invention; 

FIG. 2 Is a block diagram of a part of the image dis- 
play apparatus performing an image display proc- 
ess; 

FIG. 3 Is an Illustration of a tree structure generated 
by the image display process; 
FIG. 4 is an illustration of a 1 -dimensional map of 
the tree structure shown In FIG. 3; 
FIG. 5 is an illustration showing a relationship be- 
tween 3-dimensional display space and features 
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(form, texture, color) extracted from an image; 
FIG. 6 is an illustration sliowing a 2-dimensional 
feature space, which is divided into four parts, ac- 
cording to a second embodiment of the present in- 
vention; 

FIG. 7 is an illustration showing a 3-dimensional 
feature space, which is divided into eight parts, ac- 
cording to a second embodiment of the present in- 
vention; 

FIG. 8 is an illustration for explaining a procedure 
of acquiring each sub-space in the second embod- 
iment of the present invention; 
FIG. 9 is an illustration for explaining a procedure 
of acquiring each sub-space in the second embod- 
iment of the present invention; 
FIG. 10 is an illustration for explaining a procedure 
of acquiring each sub-space in the second embod- 
iment of the present invention; 
FIG. 11 is an illustration for explaining a procedure 
of acquiring each sub-space in the second embod- 
iment of the present invention; 
FIG. 12 is an illustration for explaining a procedure 
of distributing images to sub-spaces in the second 
embodiment of the present invention; 
FIG. 13 is an illustration for explaining a procedure 
of distributing images to sub-spaces in the second 
embodiment of the present invention; 
FIG. 14 is an illustration of a structure generated in 
the second embodiment; 

FIG. 15 is an illustration for explaining a procedure 
of generating an image display screen in the second 
embodiment of the present invention; 
FIG. 1 6 is an illustration for explaining a procedure 
of generating an image display screen in the second 
embodiment of the present invention; 
FIG. 1 7 is a flowchart of a feature-axis display space 
generation process according to the first embodi- 
ment of the present invention; 
FIG. 1 8 is a flowchart of a display-space generation 
process I according to the first embodiment of the 
present invention; 

FIG. 19 is a flowchart of a binary-tree generation 
process according to the first embodiment of the 
present invention; 

FIG. 20 is a flowchart of a display-space generation 
process II according to the first embodiment of the 
present invention; 

FIG. 21 is a flow chart of a clustering process ac- 
cording to the first embodiment of the present in- 
vention; and 

FIG. 22 is a flowchart of a clustering process ac- 
cording to the second embodiment of the present 
invention. 

FIRST EMBODIMENT 

[0021] A description will now be given of a first em- 
bodiment of the present invention. FIG. 1 is a block di- 



agram of an image display apparatus according to a first 
embodiment of the present invention. FIG. 2 is a block 
diagram of a part of the image display apparatus per- 
forming an image display process. 
[0022] In FIG. 1, the image display apparatus com- 
prises: a central processing unit (CPU) 1 which controls 
an entire operation of the image display apparatus; a 
read only memory (ROM) 2 which stores programs ex- 
ecuted by the CPU 1 ; a random access memory (RAM) 
which stores dynamic data for executing the programs 
stored in the ROM 2 and serves as a work area when 
the programs are executed; a keyboard 4 and a mouse 
5 as input devices; a monitor 6 as a display apparatus; 
an image display unit 8 which executes an image appli- 
cation 7; a peripheral interface (l/F) 9 which functions 
as an interface with a peripheral device 1 such as a dig- 
ital camera or a scanner; and a network l/F 10 which 
functions as an interface with a network 12. These com- 
ponents are connected to each other via a bus 1 3 so as 
to be controllable by the CPU 1 so that those compo- 
nents together serves as a computer to carry out various 
functions of the present Invention. 
[0023] Additionally, a memory-media driving unit 15 
such as a CD-ROM driving unit is connected to the bus 
13. The memory-media driving unit 15 reads program 
codes from a memory medium such as a CD-ROM so 
as to load the program codes stored in the memory me- 
dium to the computer so that the computer carries out 
various functions of the present invention mentioned lat- 
er. 

[0024] The image display part 8 comprises, as shown 
in FIG. 2. a feature-extraction unit 81, a clustering unit 
82, a tree-structure generation unit 83. a display-space 
generation unit 84 and a display-screen generation unit 
85. The units 81-85 perform a feature-extraction proc- 
ess, a clustering process, a tree-structure generation 
process, a display-space generation process and a dis- 
play-screen generation process, respectively, in accord- 
ance with the image application 7. 
[0025] A description will now be given of processes 
performed by the image display part 8 having the above- 
mentioned structure. The processes include 1) a fea- 
ture-extraction process. 2) a feature-space tree-struc- 
ture extraction process and 3) an image display screen 
generation process. 

1 ) feature-extraction process: 

[0026] As a feature of an image, there are various fea- 
tures such as a histogram feature, an edge feature or a 
texture feature. 

The present invention is applicable to any features. Of 
course, the present application is applicable to features 
extracted from data other than image data. For example, 
the present invention is applicable to features extracted 
from text data. Here, a description will be given of an 
extraction process of a general histogram feature. As 
the image data from which features area extracted. 
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there is image data supplied by the peripheral devices 
11 such as a digital camera or a scanner, or image data 
downloaded from the Web. There is no limitation with 
respect to the input method. 

[0027] First, a suitable color space (for example, Lab, 
Luv, HSV, etc.) is selected, and the selected color space 
is divided into a plurality of areas. Then, an investigation 
is performed as to which pixel of the image corresponds 
to which area of the color space. After counting the 
number of pixels for each area, the pixel number data 
is normalized based on the number of whole pixels. The 
normalized data of the number of pixels for each area 
corresponds to the histogram feature. The feature of a 
histogram serves as a point of the feature space of the 
histogram. As a distance between two features in the 
feature space of a histogram, a sum total of differences 
of the numbers of pixels for each corresponding area of 
the two features or a Euclid distance is generally use. 
In this way, the distance between features can be ob- 
tained. 

2) Feature-space tree-structure extraction process: 

[0028] A feature space is a high order dimension 
space, and cannot be simply displayed in 2-dimension 
on a screen. Thus, the structure of a feature space is 
first expressed by a tree-structure. Then, the feature 
space can be virtually expressed on a screen by map- 
ping the tree structure on the space of the screen. 
[0029] In order to express the feature space by a tree- 
structure, the feature space is clustered, and is divided 
into a plurality of sub-spaces so as to form nodes of the 
tree-structure. Further, each subspace (node) is clus- 
tered, and is divided into sub-spaces to form nodes. The 
tree structure of the feature space is can be generated 
by performing the above-mentioned operation recur- 
sively. That is, all features are arranged In the lowermost 
nodes, respectively, on an Individual node basis. FIG. 3 
shows an example of the thus-formed tree-structure. Al- 
though the number of nodes which divide the space may 
be any number equal to or greater than 2, the number 
of nodes here is set to 2 for the sake of convenience. In 
this tree structure, similar features are arranged close 
to each other, and a positional relationship of images on 
the image space is expressed by the tree structure. 
[0030] As for a clustering method, the general Near- 
est Neighor method, the K-average algorithm method, 
etc. can be used. Although similar images are arranged 
close to each other in this tree structure, other accura- 
cies depend on the accuracy of clustering Thus, a meth- 
od for improving the accuracy of clustering, a clustering 
according to the following procedures can be used. It 
should be noted that the dividing number is set to 2 in 
this method. 

(i) Acquisition of center point of cluster: 

a) select an arbitrary point A in a space. 



b) set the farthest point from the selected point 
A as a center point CI of the first cluster; and 

c) set the farthest point from the point CI as a 
center point C2 of the second cluster. 

5 

(il) Sort of points: 

a) select an arbitrary point P in a space. 

b) calculate distances between the selected 
10 point P and each of center points CI and C2 of 

two clusters, and obtain a difference between 
the calculated distances as a difference dis- 
tance; 

15 

difference distance = |C1-P|-|C2-P| 

c) repeat a) and b) so as to obtain the difference 
distance for all points; and 

20 d) sort all points according to ascending order 

of the difference distances. 

(iii) Division of point: 

25 [0031] A difference between difference distances of 
opposite sides of the point sorted according to the dif- 
ference distance is obtained so as to set a boundary of 
clusters between points of which distance is largest. The 
side of which difference distance is smaller than a dls- 

30 tance to the boundary of cluster belongs to C1 , and the 
side of which difference distance is larger than a dis- 
tance to the boundary of cluster belongs to C2. Moreo- 
ver, the tree structure should be well-balanced. That is, 
when it is desired to distribute images at an equal inter- 

35 val on a final display screen, the separation may be per- 
formed at the middle of the number of images. The thus- 
obtained-two clusters are set as nodes of a tree so as 
to recursively perform the same processes (1), (2) and 
(3) for each node. A tree Is generated by this operation, 

40 and finally each image belongs to a leaf node. 

[0032] With the conventional technique, a screen is 
generated from the thus-generated one tree-structure. 
However, in the present invention, a tree structure is 
generated with a meaningful feature unit, and a tree 

45 structure in the binary-tree form is generated for each 
feature. The features having such a meaning are color, 
form or texture, and an axis of a screen generated by 
the following screen generation process corresponds to 
each feature. 

50 

3) Image display screen generation process: 

[0033] After generating the tree structure, each tree 
structure is mapped in 1 -dimension. The display area 
55 on a screen is a data area for mapping in 1 -dimension. 
[0034] The tree structure is traced from a root thereof, 
and a) the data area is divided into two so that the two 
child nodes are arranged in the respective areas. The 
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area may be equally divided. In order to improve display 
accuracy, the dividing points may be decided in propor- 
tion to the number of nodes belonging to a child node 
or in proportion to a distance between a dividing point 
at the time of clustering and each center point. Further- s 
more, also in consideration of the distance of a gap be- 
tween clusters, a area corresponding to the space of the 
gap may be set and the space is not provided with a 
child node. According to such a method, the display 
screen can express further accurate similarity. 
[0035] b) Perform the above-mentioned 1 ) feature-ex- 
traction process and 2) feature-space tree-structure ex- 
traction process with respect to each child node. 
[00361 Thus, by processing recursively, all tree struc- 
tures are mapped in 1 -dimension. FIG. 4 is an example 
of mapping of a tree structure shown in FIG. 3. In this 
example, the area is equally divided while tracing the 
tree structure. 

[0037] By performing the above-mentioned process 
with respect to all tree structures, multidimensional data 
corresponding to the number of tree structures can be 
obtained. As for a screen display, since it is difficult to 
express in more than three dimensions, the data is pref- 
erably up to three dimensions. For example, if a tree- 
structure is generated based on the features of color, 
form an texture, the 3-dimensional display space having 
axes corresponding to color, form and texture shaft, re- 
spectively, can be generated. 

[0038] FIG. 5 shows an example of a final spatial 
screen display. Each point shown in FIG. 5 is a location 
of the feature which is extracted from an image accord- 
ing to parameters of form, texture and color, and the im- 
age may be displayed at that location as it is. It should 
be noted that, although form, texture and color extracted 
from an image are rendered to be features in the present 
embodiment, the present invention is not limited to such 
a feature and can be applied to any features such as a 
feature extracted from text data. 
[0039] A description will now be given, with reference 
to FIG. 17 through FIG. 21, of an image space display 
process according to the first embodiment of the present 
invention. The image space display process is per- 
formed by executing programs stored in the ROf^ 2 
while the CPU 1 uses the RAM 3 as a work area. In the 
image space display process according to the present 
embodiment, a feature Is assigned to each display di- 
mension axis, 

[0040] FIG. 1 7 is a flowchart of a process of generat- 
ing a feature-axis display space. In this process, first, a 
feature A is selected and a call is made for a display- 
space generation process I so as to generate a 1 -dimen- 
sional display space (step SI 01). Subsequently, the co- 
ordinates of the 1 -dimensional display space of the fea- 
ture A are acquired (step SI 02), and the acquired coor- 
dinates are set as coondinates of a display axis X (step 
SI 03). Additionally, a feature B is selected and a call is 
made to the display-space generation process I so as 
to generate a 1 -dimensional display space (step SI 04). 



Then, the coordinates of the 1 -dimensional display 
space of the feature B are acquired (step SI 05), and the 
acquired coordinates are set as coordinates of a display 
axis Y (step S106). Further, a feature C is selected and 
a call is made to the display-space generation process 
I so as to generate a 1 -dimensional display space (step 
S107). Then, the coordinates of the 1 -dimensional dis- 
play space of the feature B are acquired (step SI 08), 
and the acquired coordinates are set as coordinates of 
a display axis Z (step SI 09). Finally, a 3-dimensional 
display screen is generated based on coordinate values 
of the display axes acquired in the process of steps 
SI 03. S106, and S109. It should be noted that, in this 
process, the process of steps S1 01 -SI 03 makes the 
feature A to correspond to the coordinate axis X. the 
process of steps SI 04-S1 06 makes the feature B to cor- 
respond to the coordinate axis Y, and the process of 
steps SI 07-S1 09 makes the feature C to correspond to 
the coordinate axis Z. 

[0041 ] FIG. 1 8 is a flowchart of the above-mentioned 
display-space generation process I. In this process, 
when image data is input, the image feature is extracted 
to the last image (steps S201 and S202) so as to obtain 
a set of features (feature space) (step S203), Subse- 
quently, a binary-tree generation process shown in FIG. 
19 is called (step S204), and the binary-tree generation 
process is performed with respect to the set of the fea- 
tures (step S205). Then, a display-space generation 
process II shown in FIG. 20 is called based on root sets 
(nodes) of the binary tree and a display space as inputs 
(step S206), and the display space is generated by per- 
forming the display space generation process II (step 
S207). 

[0042] The routine performed at the time of execution 
of the display space generation process I shown in FIG. 
18 is shown in FIGS. 19 through 21. FIG. 19 is a flow- 
chart of a binary-tree generation process, which is called 
in step S204 and performed in step S205. In this proc- 
ess, a call is made to the clustering process shown in 
FIG. 21 so as to generate two subsets (clusters) A and 
B by half-dividing a feature set calling (step S301). The 
clustering process has been described in the description 
of 2) feature-space tree-structure extraction process. 
When the subsets A and B are generated, each of the 
subsets A and B is set as a child set (child node) of the 
feature set (step S302). Then, it is determined whether 
or not the feature element of the subset A is 1 (step 
S303), If the feature element of the subset A is 1 , it is 
determined whether or not the feature element of the 
subset B is 1 (step S304). Moreover, if the feature ele- 
ment of the subset A is not 1 in step S303, the binary- 
tree generation process is called so as to perform the 
process after step S301 by regarding the subset A as a 
set, and also perform the process of step S304. This 
process is ended if the feature element of the subset B 
is 1 in step S304. If the feature element of the subset B 
is not 1 , the binary-tree generation process is called in 
step S305 (step S306) so as to perform the process after 
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step 301 by regarding the subset B as a set, and the 
process is ended. 

[0043] FIG, 20 is a flowchart of the display-space gen- 
eration process N, which is called In step S206 and per- 
formed in step S207. In this process, it is judged whether 
or not all the features are sets (leaf sets) belonging to 
the respective lowermost nodes (step S401 ). If they are 
leaf sets, the image is arranged In the display space 
(step S402), and the process is ended. On the other 
hand, if it is judged in step S401 that the features are 
not leaf sets, the display space is divided into subsets 
A and B (step S403). Then, each of the subsets A and 
B corresponding to children of the binary tree is as- 
signed to a respective one of the sub-spaces A and B 
(step S404). Then, the display-space generation proc- 
ess II is called recursively by regarding the subset A as 
a set and the sub-space A as a display space (step 
S405). Then, the display-space generation process II is 
called recursively by regarding the subset B as a set and 
the sub-space B as a display space (step S406). The 
process of steps S403, S404 and S405 is repeated until 
a leaf set is formed, and the image is arranged in the 
display space after the leaf set is established (step 
S402). 

[0044] FIG. 21 is a flowchart of the clustering process, 
which is performed in step S301 . As described in the 
feature-space tree-structure extraction process, first, an 
arbitrary point A is selected (step S501 ), and a point far- 
thest to the selected point A is set as point C1 (step 
S502). Then, a point farthest to the point CI is set as 
point C2 (step S503), and an arbitrary point P, which has 
not been processed, is selected (step S504). Next, dif- 
ference distances between the Point P and each of the 
points CI and C2 are calculated (step S505). Then, the 
selected point P Is rendered as a point, which has been 
processed (step S506). The process of steps S504 
through S506 is repeated until unprocessed points are 
eliminated (step S507). After all points have been proc- 
essed, all points are sorted according to an ascending 
order of the difference distances (step S508). Then, a 
difference between difference distances preceding and 
proceeding each point is calculated for all sorted points 
(step S509), and the set of points is divided at a cluster 
boundary where the difference between difference dis- 
tances Is maximum (step S51 0). It should be noted that, 
although the features are assigned to the display-space 
dimensional axes as shown in FIG. 17 in the present 
embodiment, the above-mentioned display-space gen- 
eration process I may be called when the features are 
not assigned to the display dimensional axes. 

SECOND EMBODIMENT 

[0045] A description will now be given of a second em- 
bodiment of the present invention. 
[0046] An image display apparatus according to the 
second embodiment of the present invention Is the 
same as the image display apparatus according to the 



above-mentioned first embodiment. Additionally, the im- 
age display apparatus according to the second embod- 
iment has the same structure as that shown in FIG. 2, 
and description thereof will be omitted. 

5 [0047] A description will be given below of a process 
performed by the image display unit 8 according to the 
second embodiment. The process performed by the im- 
age display unit 8 includes 1) feature-extraction proc- 
ess. 2) feature-space division process and 3) image-dis- 

10 play screen generation process. 

1) Feature-extraction process: 

[0048] This processing is the same as the 1) feature- 
15 extraction process of the above-mentioned first embod- 
iment, and a description thereof will be omitted. 

. 2) Feature-space division process: 

20 [0049] A feature space is divided so as to assign a 
feature to a display space. A description will be given 
below of a dividing method of a feature space in a case 
where a 2-dimensional display space is generated. It Is 
assumed that features A and B are extracted from each 

25 image so that the feature A is assigned to the X-axis of 
the display space and the feature B is assigned to the 
Y-axis of the display space. In the case of a 2-dimen- 
sional display space, the feature space is divided Into 
four (FIG. 6). In the case of a 3-dimensional display 

30 space, the feature space is divided into eight (FIG. 7). 
[0050] In the quarter division, a positional relationship 
between the divided spaces can be accurately reflected 
to the division by following the procedure described be- 
low. 

35 [0051] It should be noted that, If the feature A of an 
image a is represented as FAa and the feature B of an 
image b is represented as FAb. a distance (0-1 ) between 
the image a and the image b In the space of the feature 
A is expressed by DA(FAa,FAb). Similarly, a distance 

-^o (0-1 ) between the image a and the image B in a space 
of the feature B is expressed by DB(FBa,FBb). The val- 
ue of the distance ranges from 0 to 1 . 

i) Acquisition of the main feature of each sub-space: 

45 

[0052] A description will be given, with reference to' 
FIGS. 8 through 11 , of a procedure for acquiring a main 
feature of each sub-space. 

[0053] First, a) an arbitrary image a is selected within 

50 a space (FIG. 8). 

[0054] Then, b) an Image that is farthest to the select- 
ed image with respect to the two features Is set as a 
main image c1 of a sub-space (lower left sub-space in 
FIG. 8). That is, the image c which maximizes the fol- 

55 lowing evaluation equation (1 ) is set as the main image 
c1. 
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DA(FAa,FAc) + DB(FBa.FBc) (1 ) 

[0055] Moreover, c) an image that is farthest to the 
image c1 with respect to the two features is set as a 
main image c2 of a sub-space (upper right sub-space 
in FIG. 9) located along a diagonal line. That is, the im- 
age c which maximizes the following evaluation equa- 
tion (2) is set as the main image c2. 

DA(FAc1 ,FAc) + DB(FBc1 ,FBc) (2) 

[0056] Furthermore, d) an image, which is close to c1 
with respect to the feature A and remote from c1 with 
respect to the feature B and remote from c2 with respect 
to the feature A and close to c2 with respect to the fea- 
ture B, is set as a main image c3 of the third sub-space 
(upper left sub-space in FIG, 10). That is, the image c 
which maximizes the following evaluation equation (3) 
is set as the main image c3. 

(1-DA(FAc1,FAc)) + DB(FBcl.FBc) -i- DA(FAc2,FAc) 
+ (1-DB(FBc2.FBc)) (3) 

[0057] Finally, e) an image, which is close to c1 with 
respect to the feature B and remote from c1 with respect 
to the feature A and remote from c2 with respect to the 
feature B and close to c2 with respect to the feature A 
and remote from 3 with respect to both the features A 
and B, is set as a main image c4 of the fourth sub-space 
(lower right sub-space in FIG. 11), That is, the image c 
which maximizes the following evaluation equation (4) 
is set as the main image c4. 

DA{FAc1 ,FAc) + (1 -DB(FBc1 ,FBc)) 
+ (1-DA{FAc2.FAG)) + DB(FBc2, FBc) 

+ DA(FAc3,FAc) + DB(FBc3.FBc) (4) 

[0058] Although the four main images are determined 
with respect to c1 , c2 and c3, it is not determined based 
on a relation with all other main images. In a case in 
which there is a margin of a process time, a much more 
accurate main image can be acquired by redetermining 
all main images based on the evaluation equation in e). 
[0059] By performing the above-mentioned process 
repeatedly until a change in a main image is eliminated, 
a further higher accuracy can be achieved. However, 
since it takes a long process time to achieve a higher 
accuracy, a selection should be made for the process to 
be used in accordance with a processing speed required 
by an application. 



ii) Distribution of images: 

[0060] The remaining images are selected one by one 
and distances to c1 through c4 are calculated so as to 
5 locate the selected image in a farthermost sub-space. 
That is, a main image cn {n=1-4), which minimizes the 
following distance, is obtained with respect to an image 
p shown in FIG. 12, and the obtained main image is lo- 
cated in a sub-space n of the main image. 

10 

DA(FAp.FAcn) + DB(FBp,FBcn) (5) 

In this way, an image can be divided into four sub-spac- 
15 es. The above-mentioned method can be expanded to 
a case of a 3-dimensional display space by assigning 
three features to each dimensional axis in the same 
manner (FIG. 13). 

[0061] Then, the above-mentioned process of i) and 
20 ii) is applied to the thus-obtained four sub-spaces. In this 
way. each sub-space can be subdivided by recursively 
processing the thus-generated sub-spaces. This proc- 
ess is repeated until only one image is located in each 
sub-space. As a result, the sub-spaces are expressed 
25 by the tree-structure as shown in FIG. 14, and the tree- 
structure extraction process of a feature space is com- 
pleted. 

3) Image display screen generation process: 

30 

[0062] In the image-display screen generation proc- 
ess, as shown in FIG. 14, the expressed tree-structure 
is traced from a root thereof, and a) a display space is 
divided into four c1-c4, as shown in FIG. 15. A simple 
35 division is used for the dividing method. It should be not- 
ed that although the equal-division has been explained 
above, there are following methods, for example. 
[0063] A method of dividing a display space while 
making the area of the display space in proportion to a 
40 number of images contained in each sub-space. 

[0064] A method of dividing a display space in propor- 
tion to a size of each sub-space which size is set as the 
maximum value of a distance between arbitrary two 
points within the sub-space. 
45 [0065] b) Each sub-space of the tree-structure is as- 
signed to a respective one of the divided display spaces. 
At this time, the positional relationship between the sub- 
spaces must be corresponded to the positional relation- 
ship, which has been taken into consideration at the time 
50 of generating the sub-spaces. If it is a sub-space of a 
leaf node of the tree-structure, the image is arranged on 
the display space since that image is assigned to the 
node. 

[0086] c) The processes of the above-mentioned a), 
55 b) of 3) are recursively performed on lower order sub- 
spaces of each sub-space. 

[0067] According to the above-mentioned operation 
the image of in the form of a tree-structure can be ar- 
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ranged in the display space, as shown in FIG. 16. How- 
ever, the distribution of the image in the thus-generated 
display space is not uniform. 

There may be a case in which an application requires a 
uniform distribution. An uneven distribution is caused by 
unevenness in the number of images assigned to each 
sub-space at the time of generating the sub-space. 
Then, a description will be given below of a method of 
assigning images to a sub-space uniformly. 
[0068] d) When assigning images uniformly to sub- 
spaces, a process of acquiring a main feature of each 
sub-space in the above-mentioned dividing process of 
a feature space described in the item 2) is the same as 
the process of the above-mentioned item 2)-i). 
[0069] However, the distribution of images in the item 
2)-ii) is performed as follows. 

1) The maximum number of images of each sub- 
Image is determined from the number M of whole 
Images in the space, and the following relationship 
is set. 

M=N/4+1 

2) An Image is selected from the remaining images 
one by one so as to calculate distances DA(FAp, 
FAcn)+DB(FBp,FBcn) to c1 through c4. The select- 
ed image is located the nearest sub-space (tempo- 
rarily referred to as A) . 

3) When the number of images which belongs to 
the sub-space A exceeds M sue to introduction of 
the distances to c1-c4 into the above-mentioned 
sub-space A, an image farthest to the main image 
within the sub-space A is removed from the sub- 
space A. Then, distances to other sub-spaces is 
calculated with respect to the removed image, and 
the removed image is located in the farthermost 
sub-space (temporarily referred to as B), However, 
when the number of images which belong to the 
sub-space B exceeds M sue to the location of the 
image in the partial space B, one image is removed 
from the sub-image B in the same manner so as to 
perform the same process with respect to sub-spac- 
es other than A and B. This process is repeated until 
the number of images becomes less than M. 

4) All images are assigned by repeating the process 
of 1), 2) and 3). 

[0070] Thus, the image is uniformly allocated to the 
display space. The present embodiment assumes that 
features are assigned to display dimensional axes and 
to achieve a higher accuracy than the first embodiment. 
Therefore, the above-mentioned feature axis display 
space generation process shown in FIG. 17 is not per- 
formed, but the process of FIGS. 18 through 21 is per- 
formed. However, the binary-tree generation process 
shown in FIG. 19 is replaced by a process shown in FIG. 



22. Therefore, in the present embodiment, the clustering 
process in step S301 is replaced by the clustering proc- 
ess shown in FIG. 22. 

[0071 ] FIG. 22 is a flowchart of the clustering process 
5 according to the second embodiment of the present in- 
vention. 

[0072] In the cluster process shown in FIG. 22, an ar- 
bitrary point A is selected first (step S601 ). Then, a point 
at which the above-mentioned evaluation equation (1) 
10 takes a maximum value with respect to the selected 
point A is set as a point c1 (step S602). Subsequently, 
a point at which the evaluation equation (2) takes a max- 
imum value is set as a point c2 (step S603). A point at 
which the evaluation equation (3) takes a maximum val- 
15 ue is set as a point c3 (step S604). A point at which the 
evaluation equation (4) takes a maximum value is set 
as a point c4 (step S605). Thereafter, an arbitrary point 
P, which has not been processed, is selected (step 
S606). Distances between the selected point P repre- 
sented by the equation (5) and each of the points c1, 
c2, c3 and c4 are calculated (step S607). Then, the point 
P is added to the cluster cn having the smallest distance 
(step S608), The selected point P is rendered to be as 
processed (step S609). All points are sorted in ascend- 
ing order of the difference distances (step S610). The 
process of steps S606 through S610 is repeated until 
unprocessed points are eliminated (step S611). Other 
processes that area not described are the same as the 
above-mentioned first embodiment. 
[0073] The present invention is not limited to the spe- 
cifically disclosed embodiments, and variations and 
modifications may be made without departing from the 
scope of the present invention. 
[0074] The present application is based on Japanese 
priority applications No. 2001 -079007 filed on March 1 9, 
2001 , No. 2001-162701 filed on May 30, 2001 , the entire 
contents of which are hereby incorporated by reference. 



1, An image-space display method of extracting fea- 
tures from images, hierarchically dividing a feature 
space of the features, virtually converting, the im- 
ages into a tree structure, dividing a display space 
accorcling to the tree-structure, and displaying the 
image space by displaying the images on each of 
the divided display spaces, 

characterized by the steps of: 

generating a tree-structure for each of the fea- 
tures; 

generating dimension data corresponding to a 
number of the features by mapping each tree 
structure in one-dimension; and 
displaying the dimension data on the corre- 
sponding divided display spaces as display co- 
ordinate-axis data. 
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8. The image space display method as claimed in 
claim 7, characterized in that when selecting a 
main image of each sub-space in the step of divid- 
ing into four sub-spaces, the main image is sequen- 

5 tially determined based on a result of calculation ac- 
cording to an evaluation equation that is previously 
established based on a distance to main image pre- 
viously determined in consideration with a position- 
al relationship with respect to each feature. 

10 

9, The image space display method as claimed in 
claim 8, characterized in that the once determined 
main image is recalculated based on evaluation 
equations based on three other main images so as 

15 to determine the main image again based on a re- 
sult of the recalculation. 



2. The image space display method as claimed in 
claim 1 , characterized in that dividing the display 
space includes the step of: 

obtaining distances between each point of the 
features and center points of two clusters with 
respect to each point; 

sorting each points according to a difference 
between the obtained distances; 
and dividing the display space into two clusters 
by setting a boundary of the clusters according 
to an order of the sorting. 

3. The image space display method as claimed in 
claim 2, characterized in that the boundary of the 
clusters is set between points of which a difference 
of distances preceding and succeeding points is 
largest in the order of the sorting. 

4. The image space display method as claimed in 
claim 3, characterized in that an area correspond- 
ing to one of dimensions of the display space is di- 
vided in accordance with distances to center points 
of the two clusters and a ratio of differences of the 
difference distances at the boundary of the clusters. 

5. The image space display method as claimed in 
claim 2, characterized in that the boundary of the 
clusters corresponds to a halfway point in the order 
of the sorting. 

6. The image space display method as claimed in 
claim 1, characterized in that the features are 
three-dimensional data corresponding to form, tex- 
ture and color of the images, and the features are 
displayed by using the three-dimensional data as 
display coordinate-axes data. 

7. An image-space display method of extracting fea- 
tures from images, hierarchically dividing a feature 
space of the features, virtually converting the imag- 
es into a tree structure, dividing a display space ac- 
cording to the tree-structure, and displaying the im- 
age space by displaying the images on each of the 
divided display spaces, 

characterized by the steps of: 



10. The image space display method as claimed in 
claim 9, characterized in that a position of the main 
20 image is calculated repeatedly until a change in the 
position does not occur, and the position at which a 
change does not occur is selected as the position 
of the main image. 

25 11. The image space display method as claimed in 
claim 7, characterized in that a number of images 
assigned to the closest main image exceeds a max- 
imum number of images containable in the sub- 
space containing the closest main image, a process 

30 of removing one of the images farthest to the main 
image from the sub-space concerned is repeatedly 
performed so as to distribute the images uniformly 
to the sub spaces. 

35 12. The image space display method as claimed in 
claim 7, characterized in that a three-dimensional 
display space is used instead of the two-dimension- 
al display space. 

40 13. An image-space display apparatus for extracting 
features from images, hierarchically dividing a fea- 
ture space of the features, virtually converting the 
images into a tree structure, dividing a display 
space according to the tree-structure, and display- 
45 ing the image space by displaying the images on 
each of the divided display spaces, 
characterized by: 

means for generating a tree-stmcture for each 
of the features; 

means for generating dimension data corre- 
sponding to a number of the features by map- 
ping each tree structure in one-dimension; and 
means for displaying the dimension data on the 
corresponding divided display spaces as dis- 
play coordinate-axis data. 

14. An image-space display apparatus for extracting 



assigning the extracted features to respective 
dimensional axes of the display space; 
dividing recursively the feature space into four 50 
sub-spaces in accordance with the dimensional 
axes while reflecting a relationship between the 
images arranged in a two-dimensional display 
space in a relationship between the images on 
the feature space; and 55 
locating the tree-structure of each sub-space in 
four sub-spaces generated by dividing the two- 
dimensional display space. 
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features from images, hierarchically dividing a fea- 
ture space of the features, virtually converting the 
images into a tree structure, dividing a display 
space according to the tree-structure, and display- 
ing the image space by displaying the images on 5 
each of the divided display spaces, 
characterized by: 

means for assigning the extracted features to 
respective dimensional axes of the display 
space; 

means for dividing recursively the feature 
space into four sub-spaces in accordance with 
the dimensional axes while reflecting a relation- 
ship between the images arranged In a two-dl- is 
mensional display space In a relationship be- 
tween the images on the feature space; and 
means for locating the tree-structure of each 
sub-space in four sub-spaces generated by di- 
viding the two-dimensional display space. 20 

15. A computer program comprising program code 
means that, when executed on a computer, in- 
structs the computer to perform the method of any 
oneof claims 1 to 12. 25 
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(54) Image space display method and apparatus 

(57) An image space display method facilitates a us- 
er to grasp a feature space by assigning each feature 
to a respective one of dimensional axes of a display 
space. The image-space display method extracts fea- 
tures from images, hierarchically divides a feature 
space of the features, virtually converts the images into 
a tree structure, divides a display space according to the 



tree-structure, and displays the image space by display- 
ing the images on each of the divided display spaces. 
In the method a tree-structure is generated for each of 
the features. Dimension data corresponding to a 
number of the features is generated by mapping each 
tree structure in one-dimension. The dimension data is 
displayed on the corresponding divided display spaces 
as display coordinate-axis data. 
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